1. Field of the Invention
The present invention relates, in general, to methods and systems for testing and qualifying software tools and applications such as those used to provide hardware and/or software service products including network-based monitoring applications, and, more particularly, to methods automating the testing and qualification of applications, tools, routines, and the like such as software programs or plug-ins/add-ons developed to provide new hardware or software serviceability or other features to network monitoring tools/applications that may be provided via a network delivery model.
2. Relevant Background
Software testing is used to provide developers and purchasers of a new product or features being added to an existing software product with information about the quality and effectiveness of the product or service. The product is preferably tested or qualified for use in the context in which it is intended to operate such as within a similar computer system or network with similar operating conditions including similar running software applications, similar hardware devices such as servers, data storage devices and configurations, and communications components or networking devices. Customers or users of new products want assurances that a new product or upgrades to an existing software service have been adequately tested and will not cause their system to crash or even result in degradations in the system or device operations.
Testing cannot completely establish the correctness of a new software product or added feature of an existing application, but it instead typically provides a comparison of the state and behavior of the software product with standards such as comparable products, prior versions of the product, and customer or user expectations for the product. The primary purpose of testing is to detect software failures so that defects may be uncovered and corrected, but this has become more difficult with the increasing complexity of software products and the systems or networks in which the software products are used. Testing or qualification of software products or added features is also used to determine whether the product or new features are compatible with other applications, operating systems, and hardware of the system running the product or using the new features.
Testing or qualifying software products is further complicated when the software product is adapted to provide services over or within a network (e.g., services are provided based on the network delivery model). It is often difficult to replicate a typical customer's or user's data center or distributed computer network in a controlled or laboratory setting, e.g., a test lab with identical computer devices, similar storage configurations, and networking devices linking these components including running applications and operating systems. The option of running the new software product on the customer's system for qualification is also generally not an option as the user wants the product to be tested or proven before loading onto their system.
The information technology industry has, for example, developed management software or suites of application that may be used to monitor and manage operation of a data center, data storage array or system, or other computer system. For example, Sun Microsystems, Inc. provides array management software including an Auto Service Request (ASR) feature or module that monitors the array system health and performance (e.g., based on telemetry collected operating data from the customer's system components) and automatically provides notifications when critical events occur (e.g., generates an automatic Service Request case upon a critical alarm that is transmitted to a support or monitoring center). Such monitoring and management tools may be thought of as providing support services or serviceability features via a network or by a network delivery model, and these tools have been very popular with owners of large computer systems such as data centers for monitoring operation of storage devices and even for monitoring and controlling power usage. A drawback of such network delivered services is the complexity and difficulty of testing new products or even adding new features or services to these management applications.
Providing product support services for servers, storage, software, and so on requires a qualification process for each product or added feature prior to delivery. The new hardware or software “serviceability” features or service products may be provided across such attributes as power consumption, component faults, component or environment monitoring, or even user-specified operating parameters. Presently, there is little or no automation around the network service delivery software that is written to facilitate testing of remote services or features such as automated case creation, power analysis predictions, and so on that may monitor using telemetry or the like. As a result, existing testing and qualification is an expensive and time-consuming process that may even produce incomplete results (e.g., when a user's system is not adequately replicated in a lab setting).
There remains a need for methods for qualifying network-delivered services and other similar software products. Preferably such a qualification method would address the need to test or qualify network-delivered services that may include automated service request creation or other monitoring outputs/alarms/alerts such as those created upon observation of hardware component faults, power monitoring, event monitoring, and other monitoring of storage, servers, and service offerings such as MySQL, GlassFish, and the like. For example, it is desirable to provide a qualification method for speeding up the process of providing new hardware or software serviceability features for a network monitoring or similar application that reduces field costs and delays in delivery of support offerings to the marketplace).